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OVERVIEW  OF  ADVANCED  SYSTEMS  DIVISION 
CRITERION  RESEARCH  (MAINTENANCE) 


L INTRODUCTION 

The  Advanced  Systems  Division  (AS)  of  Air 
Force  Human  Resources  Laboratory  (AFHRL)  has 
had  two  separate  and  distinct  criterion  R&D 
programs  -one  concerning  pilot  performance,  and 
the  other  concerning  maintenance  performance. 
Today  I am  addressing  our  maintenance  program. 

Maintenance  of  hardware  is  currently  an  ex- 
tremely costly  operation  for  the  Department  of 
Defense  (DOD).  High  maintenance  cost  is  the 
primary  cause  of  high  systems  ownership  cost.  For 
some  electronic  maintenance  specialities,  nearly 
one  year  of  broad  formal  training  is  given  first 
enlistment  personnel.  And  maintenance  training 
generally  is  long  and  costly.  Even  with  such 
lengthy  training,  the  efficiency  of  maintenance 
could  be  greatly  improved.  Improved  job  instruc- 
tions and  information,  as  well  as  increased  use  of 
job  (task)  oriented  training  have  great  potential  for 
decreasing  maintenance  training  time  and  im- 
proving the  job  performance  of  maintenance  tasks. 

But,  to  maximize  such  potential  and  to  ensure 
more  efficient  maintenance,  the  criteria  for  the 
selection,  training,  assignment,  and  promotion  of 
maintenance  men  should  be  the  demonstrated 
ability  of  maintenance  personnel  to  perform  the 
tasks  of  their  jobs.  To  enforce  such  criteria,  the 
key  job  tasks  must  be  identified  and  the  ability  to 
perform  identified  tasks  must  be  ascertained.  Since 
the  ability  to  perform  many  or  most  of  the  identi- 
fied tasks  will  not  be  part  of  the  normal  repertoire 
of  those  being  selected  for  jobs,  appropriate  action 
must  be  taken  to  develop  the  ability  to  perform 
job  tasks.  Of  course,  these  actions  arc  “easier  said 
than  done.” 


II.  THE  CRITERION  PROBLEM 

If  we  can  produce  a measuring  device  that 
actually  measures  the  ability  to  perform  the  de- 
sired behaviors  under  all  the  desired  conditions,  we 
have  an  ultimate  criterion  measure.  But  the  fact, 
that  we  usually  cannot  develop  such  a device, 
forces  us  to  settle  for  a secondary  criterion 
measure  which  is,  at  best,  somewhat  different  than 
the  ultimate.  As  we  see  it,  this  difference  between 
the  real  world  and  the  simulation  of  the  real  world 
(for  testing  purposes)  is  the  criterion  problem. 

A common  example  of  such  a criterion  problem 
presents  itself  when  we  attempt  to  measure  an 


individual’s  ability  to  drive  automobiles.  To 
measure  such  ability  completely  we  would  have  to 
devise  a test  that  would  measure  his  ability  to  per- 
form all  driving  tasks  of  all  automobiles,  on  all 
types  of  roads,  in  all  traffic  conditions,  under  all 
types  of  weather  conditions,  whether  he  is  being 
observed  or  not.  It  is  obvious  that  it  would  be 
virtually  impossible  to  meet  all  of  these  conditions 
under  practical  testing  conditions.  We,  therefore, 
settle  for  a less  rigorous  test  criterion.  We  assume 
that  he  can  drive  any  automobile  adequately,  if  he 
demonstrates  in  a performance  test  that  he  can 
perform  most  driving  tasks  in  one  automobile,  in 
normal  traffic,  while  being  observed. 

But  many  times,  it  is  inconvenient  and  con- 
sidered too  costly  to  administer  even  such  a driver 
performance  test  and  an  attempt  is  made  to 
develop  a paper-and-pencil  test  which  will  deter- 
mine that  an  individual  can  drive  adequately.  But 
such  a test  cannot  be  considered  to  be  a valid  sub- 
stitute, unless  a high  empirical  relationship  to  the 
criterion  measure  can  be  demonstrated.  In  the 
practical  world  of  test  development,  the  driver  per- 
formance test  would  be  considered  an  adequate, 
near  ultimate  criterion  test  for  validation  of  such  a 
paper-and-pencil  substitute.  Many  times  such  a 
paper-and-pencil  test  is  used  without  being  vali- 
dated against  such  a near  ultimate  criterion  test. 
The  use  of  such  an  unvalidated  test  would  be  an 
extremely  dangerous  practice,  since  it  is  assumed 
by  most  users  that  it  measures  an  individual’s 
ability  to  drive,  when  in  fact,  we  are  not  sure  what 
it  is  measuring. 

This  criterion  problem  has  long  plagued 
measurement  theorists  and  practitioners,  as  well  as 
curriculum  researchers.  The  use  of  job  tasks,  and 
performance  examinations  based  on  these  tasks  as 
near  ultimate  criteria  for  evaluation  of  selection 
devices,  was  fust  emphasized  as  a result  of  the 
work  of  Army  and  Navy  measurement  psycho- 
logists during  World  War  II.  In  1946,  Jenkins 
discussed  the  problem  in  light  of  the  experiences 
of  Navy  psychologists  in  an  article  in  the  American 
Psychologist,  entitled  “Validity  for  What?” 

Psychologists  in  general  tended  to  accept 
the  tacit  assumption  that  criteria  were  either 
given  of  God  or  just  to  be  found  lying 
about. . . . The  novice  of  1940,  searching 
through  many  textbooks  and  much  journal 
literature  would  have  been  led  to  conclude 
that  expcndiency  dictated  the  choice  of  cri- 
teria and  that  the  convenient  availability  of 
a criterion  was  more  important  than  its 
adequacy. 
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In  1964,  the  late  Rains  Wallace  presented  a paper 
at  the  annual  convention  of  American  Psycho- 
logical Association  (APA),  which  also  appeared  in 
the  American  ftychologist  (Wallace,  1965a).  It 
indicated  that  much  of  what  Jenkins  said  in  1946 
was  still  true. 

In  the  18  years  which  have  followed,  we 
have  become  wiser  and  sadder  about  the  cri- 
terion problem.  If  we  have  not  accom- 
plished a great  deal,  if  we  tend  to  use  the 
expedient  criterion  with  the  comforting 
thought  that  some  day  we  will  get  down  to 
constructing  better  ones,  if  we  concentrate 
on  criteria  that  are  predictable  rather  than 
appropriate,  we  do  operate  with  varying 
levels  of  guilt  feelings.  We  have  not  done 
much  about  it,  but  we  know  we  should. 

In  1965,  Wallace  presented  another  paper,  in 
which  he  addressed  the  criterion  problem  very 
succinctly  as  it  applies  to  electronic  maintenance. 

All  of  this  is  prelude  to  my  main  thesis 
which  is  in  no  sense  revolutionary,  original, 
or  controversial.  I state  it  because  it  is 
honored  in  the  breach.  It  is  that  the  nature 
of  our  proficiency  measures  determines  how 
we  select,  classify,  train,  maintain  and  assess 
our  human  resources.  If  the  measures  are 
largely  irrelevant  to  the  jobs  we  want  done, 
we  will  select  the  wrong  men,  classify  them 
incorrectly,  and  train  them  wrong.  This  is 
true  because  these  proficiency  measures  are, 
or  should  be,  the  criteria  against  which  we 
validate  our  selection  and  classification  pro- 
cedures and  evaluate  our  training  content 
and  methodology  or  our  supervisory  tech- 
niques. Thus,  if  I use  a test  of  advanced 
electronics  theory  as  the  proficiency 
measure  for  electronics  maintenance  and  as 
the  criterion  against  which  to  evaluate  a test 
for  selecting  men  to  go  into  maintenance 
training.  I will  end  up  choosing  a selection 
test  which  rejects  men  who  are  not  well 
above  average  in  both  reading  and  arithme- 
tic ability.  In  the  process  I might  reject  a 
great  many  who  are  outstanding  in  their 
ability  to  get  their  hands  on  a piece  of 
machinery  and  make  it  work.  1 might  also 
accept  a number  who  (like  myself)  arc  so 
lacking  in  the  simplest  manipulative  ability 
that  their  hands  could  have  been  cut  off  at 
the  wrists  at  birth  without  seriously 
affecting  their  outputs.  So,  when  I decided 
what  proficiency  measures  to  use,  I also 
decided  what  kind  of  men  I was  going  to 
put  into  training  for  the  job. 

But  it  doesn’t  end  there,  l or  when  1 now 
approach  the  problem  of  how  to  train  men 
to  perform  the  tasks  involved  in  the  job,  1 
must  make  decisions  about  what  should  be 
taught  and  what  methods  should  be  used  in 
teaching  it.  The  only  way  I have  of  reaching 
such  decisions  (except  by  divination  which 
is,  admittedly,  not  a rare  procedure)  is  to 
measure  and  compare  the  performances 


achieved  with  various  curricula  and 
methodologies.  So,  in  the  case  of  the  elec- 
tronics maintenance  course,  1 put  in  lots  of 
reading  about  electronics  theory  and  I 
produce  graduates  who  can  read  and  write 
electronics  theory  while  their  equipment 
deteriorates  in  hopeless  inoperativeness 
(Wallace.  1965  b,  p.4). 

Influenced  in  part  by  the  above  statement,  we 
at  the  Advanced  Systems  Division  decided  to  do 
something  about  the  criterion  problem  as  it 
applied  to  maintenance.  And  although  our  work 
was  at  times  delayed  and  sidetracked,  twelve  years 
later  we  do  have  some  R&D  completed  which  we 
can  talk  about.  However,  the  grim  and  vivid 
picture  that  Rains  Wallace  painted  in  1965  is  still 
true  for  most  of  the  operational  Air  Force. 

Our  approach  to  the  criterion  problem  has  been 
to  study  and  analyze  both  measurement  literature 
and  maintenance  jobs,  and  to  develop  job  task 
performance  tests  (JTPT)  for  key  maintenance 
tasks  which  were  selected  on  the  basis  of  these 
analyses.  We  developed  these  JTPT  to  be  as  near  to 
ultimate  job  criteria  as  possible  in  keeping  with  the 
following  suggestion  of  Frederiksen: 

The  objective,  presumably,  is  to  get  as  close 
as  is  feasible  to  the  ultimate  criterion;  but  as 
has  just  been  seen,  when  one  gets  too  close 
to  the  teal-fife  situation,  control  of  the  con- 
ditions for  adequate  observation  is  lost. 

Observation  of  real-life  behavior  is 
ordinarily  not  a suitable  technique  for 
measurement.  The  type  of  measure  that  is 
recommended  for  first  consideration  in  a 
training  evaluation  study  is  the  type  which 
most  closely  approximates  the  real-life  situa- 
tion, that  which,  in  this  chapter,  has  been 
called  eliciting  lifelike  behavior.  If  it  is  not 
feasible  to  wait  for  the  behavior  to  happen 
in  real  life,  then  lifelike  occasions  can  be 
provided  for  the  behavior  to  occur  in  a test 
situation  (Frederiksen.  1962,  p.334). 

Admittedly,  an  examination  made  up  of  tasks 
removed  from  their  actual  job  environment  is  not 
an  ultimate  criterion  test.  Under  actual  job  situa- 
tions, the  graduate  may  have  to  perform  these 
tasks  in  cramped  quarters;  under  stresses  of  time, 
noise,  heat,  or  cold;  or  with  an  excited  boss  inter- 
fering. These  conditions  of  stress  are  usually  not 
constant  variables,  but  change  from  day-to-day 
and  from  hour-to-hour.  The  assumption  usually 
has  to  be  made  that  the  individual  can  perform  a I 

task  under  conditions  of  stress,  provided  he  can 
perform  the  same  task  well  under  normal  con- 
ditions. A formal  performance  examination  has  its 
own  set  of  stresses,  which  may  not  be  the  same  as 
job  stresses,  but  their  presence  may  tend  to  offset 
the  lack  of  job  stresses.  Formal,  job  task  perform- 
ance examinations  are  the  closest  usable 
simulation  of  the  real  maintenance  jobs  presently 
available.  They  are  far  better  than  no  performance 
tests  at  all. 


6 


III.  REVIEW  OF  PERFORMANCE  MEASUREMENT 
(PM)  LITERATURE 

In  regard  to  the  literature  reviews  and  analyses 
made  for  PM  (Foley,  1967,  1974),  many  valuable 
PM  efforts  have  been  reported  by  the  Army,  Navy, 
and  Air  Force.  Flowever,  most  of  these  efforts 
have  not  been  systematic  efforts,  having  as  their 
prime  objective  the  improvement  of  the  state-of- 
the-art  of  PM.  Rather,  they  have  been  ad  hoc  PM 
developments  to  support  job  oriented  training 
research  programs.  A notable  exception  was  the 
work  of  the  Air  Force  Personnel  and  Training 
Research  Center  (AFPTRC)  Maintenance  Labora- 
tory. (Another  more  recent  systematic  Army 
effort,  accomplished  by  the  Human  Resources 
Research  Organization  (HumRRO)  was  not 
covered  in  these  reviews  (Vineberg  et  al„  1970a, 
1970b;  Vineberg  & Taylor.  1972a,  1972b).  As  to 
civilian  R&D,  during  the  initial  PM  literature 
review  (Foley,  1967),  a serious  attempt  was  made 
to  identify  and  include  the  results  of  PM  R&D 
from  the  civilian  vocational  education 
establishment.  None  was  found. 

A substantial  outcome  of  the  review  of  other 
PM  efforts  was  a consolidation  of  research  results 
concerning  the  correlations  between  results  of  PM 
for  various  maintenance  tasks  and  paper-and-pencil 
theory  tests,  job  knowledge  tests,  and  school 
marks.  As  to  their  value  for  measuring  ability  to 
perform  maintenance  tasks,  this  research  evidence 
gives  a low  rating  to  all  of  these  paper-and-pencil 
based  measures  of  school  and  job  success.  Table  1 
shows  correlations  that  have  been  obtained  by 
comparing  JTPT  to  theory  tests,  and  to  job- 
knowledge  tests.  The  latter  two  are  paper-and- 
pencil  tests.  Table  1 also  includes  correlations  of 
JTPT  with  school  marks.  As  indicated  earlier, 
school  marks  have  been  heavily  weighted  with  the 
papcr-and-Dencil  test  scores.  An  examination  of 
this  table  indicates  that  the  correlations  of  JTPT 
scores  with  theory  test  scores  are  generally  some- 
what lower  than  with  job-knowledge  tests.  None 
of  these  measures  is  sufficiently  valid  for  use  as 
substitutes  for  JTPT  (Foley,  1967,  1974). 

The  personnel  system,  which  includes  formal 
training,  depends  almost  exclusively  on  such 
paper-and-pencil  tests  for  making  initial  selection, 
for  ascertaining  effectiveness  of  training  and  for 
the  promotion  of  maintenance  personnel.  The 
effectiveness  of  formal  training  for  the  mechanical 
maintenance  specialties  is  measured  mainly  by 
scores  obtained  from  such  paper-and-pencil  job 
knowledge  tests,  even  though  the  students  in  these 
training  programs  have  received  at  least  some 
“hands-on”  practice  on  many  mechanical  mainte- 


nance tasks.  The  measures  of  effectiveness  of 
formal  training  programs  for  the  electronic  mainte- 
nance specialties  include  scores  from  paper- 
and-pencil  job  knowledge  tests,  as  well  as  theory 
tests.  Students  in  these  electronic  maintenance 
courses  receive  little  if  any  “hands-on”  practice  in 
their  maintenance  tasks. 

The  selection  tests  for  both  mechanical  and 
electronic  maintenance  specialties  have  been  stand- 
ardized against  composite  scores  from  paper- 
and-pencil  tests.  This  means  that  the  people 
selected  for  the  maintenance  specialties  have  been 
selected  not  on  their  aptitude  for  performing  the 
tasks  of  their  maintenance  jobs,  but  on  their  apti- 
tude for  making  high  scores  on  paper-and-pencil, 
theory  and  job  knowledge  tests. 

The  specialty  knowledge  test  (SKT)  and  the 
promotion  fitness  examination  (PFE)  used  for 
advancement  up  the  maintenance  career  ladders 
also  are  paper-and-pencil  job  knowledge  tests.  At 
the  present  time,  throughout  his  whole  career,  a 
maintenance  specialist  is  not  required  to  demon- 
strate on  formal  JTPTs  that  he  can  efficiently  and 
effectively  perform  the  tasks  of  his  job. 

IV.  THE  MAN-MACHINE  INTERFACE 
FOR  MAINTENANCE 

The  maintenance  R&D  supported  by  AS  has 
emphasized  the  man-machine  interface.  From  this 
point  of  view,  PM  for  all  personnel  associated  with 
machine  systems  must  determine  the  ability  of 
such  personnel  to  perform  tasks  generated  by  the 
man-machine  interface.  Although  there  may  be 
some  overlap,  most  of  the  task  functions  de- 
manded by  a machine  system  of  its  operator 
personnel  arc  different  from  those  task  functions 
demanded  of  its  maintenance  personnel.  Herein, 
lies  most  of  the  unique,  distinguishing  character- 
istics of  PM  for  maintenance.  As  a result,  this 
section  of  my  paper  will  be  devoted  to  a discussion 
of  the  complexity  of  maintenance  task  functions. 

Past  Human  Factors  Emphasis 

But  before  discussing  the  characteristics  of  task 
functions  for  maintenance,  it  might  be  well  to  call 
attention  to  the  fact  that  human  factors  establish- 
ments have  given  much  more  attention  to  the 
operator  interface  with  machines  than  to  the 
maintenance  personnel  interface.  Many  actions  arc 
taken  to  maximize  effective  and  efficient  perform- 
ance of  the  operator.  Work  stations  are  human 
engineered  to  maximize  the  efficiency  and  com- 
fort of  the  human  operator.  Major  training 
facilities  are  provided,  so  that,  operators  can 


Table  / Correlations  Between  Job-Task  Performance  Tests  and  Theory 
Tests.  Job  Knowledge  Tests,  and  School  Marks 


Type  of  Job  Task 
Performance  Test  (JTPT) 

Theory 

Tests 

Job  Knowl- 
edge Tests 

School 

Marks 

Anderson 

Test  Equipment  JTPT 

18  .33 

(19621 

Evans  and 

Troubleshooting  JTPT 

24  & 36 

12  & 10 

35 

Smith  (1953) 

Mackie  el 

Troubleshooting  JTPT 

38 

39 

at..  (1953) 

Saupe  (1955) 

Troubleshooting  JTPT 

55 

56 

Brown  et  al.. 

Troubleshooting  JTPT 

40 

(1959) 

Test  Equipment  JTPT 

29 

Alignment  JTPT 

28 

Repair  Skills  JTPT 

19 

Williams  and 

Troubleshooting  JTPT 

Whitmore 

(Inexperienced  Subjects) 

23 

(1959) 

(Experienced  Subjects) 

15 

Adjustment  JTPT 

(Inexperienced  Subjects) 

.02 

(Experienced  Subjects) 

.21 

Acquisition  Radar  JTPT 

(Inexperienced  Subjects) 

.03 

36 

(Experienced  Subiects) 

14 

22 

Target  Tracking  Radar  JTPT 

(Inexperienced  Subjects) 

24 

33 

(Experienced  Subjects) 

.20 

38 

Missile  Tracking  Radar  JTPT 

(Inexperienced  Subjects) 

09 

15 

(Experienced  Subjects) 

19 

32 

Computer  JTPT 

(Inexperienced  Subjects) 

08 

24 

(E  xpenenced  Subiects) 

06 

14 

Total  JTPT 

(Inexperienced  Subjects) 

14 

(Experienced  Subiects) 

20 

Crowder  et 

Troubleshooting  JTPT 

11 

18  - 32 

al..  (1954) 

receive  a large  amount  of  supervised  practice  in 
performing  typical  tasks  of  their  job.  Graduation 
from  training  is  based  primarily  on  demonstrated 
ability  to  perform  job  tasks.  And,  periodic  checks 
are  made  of  the  operator’s  ability  to  perform  the 
critical  tasks  of  his  job.  These,  of  course,  are  not 
ail  of  the  many  efforts  made  to  maximize  the 
performance  of  human  operators. 

Generally,  the  human  factors  establishment  has 
given  little  attention  to  the  effectiveness  and 
efficiency  of  the  maintenance  man’s  interface  with 
hardware.  The  maintenance  work  of  AS,  including 
the  PM  work,  has  emphasized  this  neglected  inter- 
face, but  typically,  this  part  of  our  program  has 
received  little  management  visibility  or  support. 

The  Structure  of  the  Man-Machine 
Interface  for  Maintenance 

One  of  the  results  of  our  R&D  for  maintenance 
has  been  the  evolution  and  articulation  of  a struc- 
ture for  handling  maintenance  functions  and  their 
complex  relationships  in  a systematic  manner.  This 
structure  includes  (a)  standard  maintenance  func- 
tions and  action  verbs,  (b)  a working  definition  of 
a maintenance  task,  and  (c)  schemes  for  handling 
the  complexities  of  maintenance  tasks. 


Standard  Maintenance  Functions  and  Action 
Verbs.  The  establishment  of  standard  maintenance 
functions  and  action  verbs  has  been  one  of  the 
widely  accepted  results  of  the  Air  Force  Systems 
Command’s  (AFSC)  job  performance  aids  (JPA) 
effort  entitled  “Presentation  of  Information  for 
Maintenance  and  Operation”  (PIMO).  (Although 
the  PIMO  project  was  managed  by  the  Space  and 
Missile  Systems  Organization  (SAMSO)  of  AFSC, 
AS  provided  active  participation  and  technical 
inputs  during  the  entire  project  from  1966 
through  1969  AS  has  incorporated  the  key 
findings  and  outputs  of  PIMO  in  its  own  JPA 
efforts).  Early  in  the  PIMO  project,  it  was  found 
that  many  maintenance  action  verbs  and  functions 
were  used  by  maintenance  people,  some  with 
several  different  meanings.  Part  of  this  confusion 
was  caused  by  the  language  used  in  maintenance 
technical  orders  which  were  written  by  different 
people  and  produced  by  many  different  hardware 
manufacturers.  As  a result,  maintenance  techni- 
cians themselves  did  not  generally  use  precise 
language.  A study  was  made  to  identify  and  define 
these  action  verbs.  Where  two  or  more  verbs  were 
used  to  indicate  a similar  action,  the  preferred  verb 
was  selected  based  on  the  expressed  preferences  of 
a sample  of  maintenance  men  with  a wide  range  of 
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maintenance  Air  Force  specialty  code  (AFSC). 
The  use  of  the  preferred  verbs  of  this  list  is  now  a 
firm  requirement  of  Air  Force  technical  order 
specifications,  as  well  as  of  recent  Army  and  Navy 
specifications  (see  Joyce,  Chenzoff,  Mulligan,  & 
Mallory,  1973,  pp.  97-142). 

A Working  Definition  of  a Maintenance  Task. 
Within  this  list  of  action  verbs  are  a number  of  key 
action  verbs  (functions).  A key  action  verb,  with 
an  appropriate  specific  hardware  unit  as  its 
predicate,  becomes  a task  statement.  Such  a task 
statement  represents  a maintenance  task  which  can 
be  demanded  by  the  existence  and  operation  of  a 
specific  machine  subsystem.  A list  of  these  func- 


tion.' is  found  in  AFHRL-TR-7343(I)  (Joyce  et 
al.,  1973.  pp.  19-20).  This  list  includes  functions 
which  are  found  in  both  mechanical  and  electronic 
jobs.  Some  apply  to  only  mechanical  jobs  and 
some  apply  to  both. 

Schemes  for  the  Systematic  Consideration  of 
Maintenance  Functions  and  Tasks.  Three  schemes 
have  been  developed  for  the  systematic  considera- 
tion of  maintenance  functions  and  tasks,  and  the 
key  factors  that  affect  them. 

Scheme  One.  A convenient  model  for  cate- 
gorizing these  maintenance  functions  with  relation 
to  the  type  of  hardware  and  the  level  of  main- 
tenance is  presented  in  Figure  1.  The  common 
maintenance  functions  already  mentioned  together 


Figure  I.  A functional  representation  of  the  DOD  maintenance  structure 
(shaded  portion  indicates  scope  of  AFHRL  PM  development  for  electronic  maintenance). 


with  use  usage  of  test  equipment  and  handtools 
are  represented  on  one  axis  of  the  model.  Since 
mechanical  and  electronic  subsystems  usually 
require  a different  variety  of  maintenance  actions, 
they  are  represented  by  another  axis.  (In  regard  to 
this  axis,  mechanical  maintenance  could  be  further 
divided  into  two  categories,  one  represented  by 
hardware  such  as  jet  engines  and  another , by 
hardware  such  as  airframes,  and  tank  and  ship 
hulls). 

The  third  axis  of  the  model  represents  the  three 
levels  or  categories  of  maintenance  now  found  in 
the  military  services.  Organizational  maintenance 
is  the  first  level.  It  is  usually  aimed  at  checking  out 
a whole  machine  subsystem  and  correcting  any 
identified  faults  as  quickly  as  possible.  Flight  line 
maintenance  falls  in  this  category.  A system  is 
checked  out.  If  it  does  not  work,  the  line  replace- 
able unit  (LRU)  or  “black  box”  causing  the  mal- 
function is  identified  and  replaced.  This  major 
component  is  then  taken  to  the  field  shop  (inter- 
mediate maintenance)  where  it  is  again  checked 
out  and  the  faults,  authorized  for  correction,  are 
corrected.  The  corrective  actions,  authorized  at 
the  intermediate  level,  vary  greatly  from  system  to 
system  depending  on  the  maintenance  concept  of 
each  system.  On  some  systems,  the  maintenance 
man  will  troubleshoot  the  “black  box”  to  the 
piece  part  level.  In  more  modern  equipment,  he 
will  identify  a replaceable  module  made  up  of 
many  piece  parts.  Some  modules  are  thrown  away, 
others  sent  to  the  depot  for  repair.  Any  line  re- 
placeable units  which  the  field  shop  are  unable,  or 
unauthorized,  to  repair  are  sent  to  the  depot  for 
overhaul. 

Organizational  and  intermediate  level  organiza- 
tions are  manned  primarily  by  enlisted  technicians 
whose  average  length  of  service  is  rather  short 
(slightly  more  than  4 years  in  the  Air  Force). 
Depots  are  manned  largely  by  civilian  personnel 
with  a much  higher  level  of  experience  and  longer 
retention  time.  Using  this  model,  it  has  been 
possible  to  specify  areas  of  concentration  for 
study. 

Since  PM  requirements  for  maintenance  are  so 
different  for  the  various  blocks  indicated  in  this 
model,  it  is  extremely  important  that  PM  re- 
searchers indie  te  the  precise  blocks  of  their  con- 
centration. To  date,  AS  has  concentrated  on  the 
shaded  electronic  portions  of  this  model  (Figure 
1).  The  resultant  model  battery  of  48  JTPT 
together  with  their  symbolic  substitutes  will  be 
described  later.  In  addition,  a battery  of  eleven 
JTPT  was  developed  on  an  ad  hoc  basis  (Shriver  & 
Foley,  1975)  for  mechanical  tasks  at  the  organiza- 
tional level  of  maintenance  (see  shaded  portion  of 


Figure  2).  The  HumRRO  work,  mentioned  pre- 
viously (Vineberg  et  al.,  1970a,  1970b;  Vineberg  & 
Taylor,  1972a,  1972b)  was  concerned  with  me- 
chanical hardware  (tank  and  truck).  The  thirteen 
tests  developed  concerned  the  maintenance  func- 
tions which  are  indicated  by  the  shaded  portions 
of  Figure  3. 

Scheme  Two.  Maintenance  functions  have 
limited  meaning  unless  applied  to  specific  hard- 
ware. A task  identification  matrix  (TIM)  is  an  ex- 
tremely effective  and  necessary  device  for  inter- 
facing these  maintenance  functions  with  the 
appropriate  hardware  units  and  thus  identifying 
the  maintenance  tasks  that  are  generated  by  a 
specific  machine  subsystem  (see  Figure  4).  The 
TIM,  when  properly  structured,  will  reflect  the 
maintenance  level  or  levels  of  interest,  that  is 
organizational,  intermediate  and/or  depot. 
AFHRL-TR-73  43(I)  (Joyce  et  al.,  pp.  16-37) 
provides  detailed  directions  for  developing  a TIM. 

Scheme  Three.  A matter  of  serious  concern 
when  developing  and  structuring  PM  for  mainte- 
nance tasks  is  the  interaction  among  the  main- 
tenance tasks  for  one  hardware.  A four-level 
hierarchy  of  dependencies  can  be  stated.  Figure  5 
gives  a graphic  presentation  of  these  dependencies 
among  maintenance  activities  for  an  electronic 
hardware. 

The  checkout  of  the  AN/APN-147  (Doppler 
Radar),  for  example,  can  be  a task  in  its  own  right. 
But  the  same  checkout  activity  becomes  an  ele- 
ment of  other  major  tasks  such  as  calibrate.  The 
calibration  of  doppler  radar  includes  the  operation 
of  specific  general  and  special  test  equipments,  the 
use  of  specific  hand  tools,  as  well  as  the  checkout 
activity.  Troubleshooting  of  an  electronic  equip- 
ment; such  as  AN/APN-147,  requires  the  use  of 
general  and  special  test  equipments.  It  may  require 
remove  and  install  activities  and/or  adjust,  align, 
and  calibrate  activities.  Efficient  troubleshooting 
practice  usually  requires  the  use  of  a cognitive 
strategy  to  adequately  track  the  dependent 
activities  (but  the  cognitive  strategy  in  itself  is  not 
troubleshooting).  Any  troubleshooting  task  should 
begin  and  end  with  an  equipment  checkout. 
Because  of  these  various  and  varying  dependency 
relationships,  such  activities  as  checkout,  remove, 
install,  disassemble,  adjust,  align,  calibrate,  or 
troubleshoot  cannot  legitimately  be  considered  as 
discrete  tasks,  even  for  one  electronic  system . 

Another  confounding  factor  is  the  false  cor- 
respondence that  the  same  functional  verbs  create 
when  applied  to  different  electronic  hardware.  For 
example,  personnel  with  the  Avionic  Inertial  and 
Radar  Navigation  Systems  Specialist,  AFSC 
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Figure  4.  Example  of  a task  identification  matrix  (TIM).  Cell  entries:  — (dash)  no  maintenance  task 
of  this  type  is  performed  on  this  hardware  item;  0 - task  of  type,  performed  at  organizational 
level;  I - task,  performed  at  intermediate  level;  and  D - task,  performed  at  depot  level. 
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Figure 


Troubleshoo  t 

5.  Indicating  the  dependencies  among  maintenance  functions 
for  electronic  hardware  ( functions  italicized). 


328X4,  are  maintaining  at  least  SO  major  elec- 
tronic subsystems.  Many  vintages  of  hardware 
design  are  represented.  The  checkout  activity  for 
each  is  different  fboth  in  content  and  difficulty) 
and  in  some  cases,  very  different.  The  lack  of 
correspondence  of  alignment,  calibration,  and 
troubleshooting  tasks  from  one  specific  equipment 
to  another  is  even  greater.  An  example  of  the  lack 
of  correspondence  from  one  hardware  to  another 
is  the  wide  difference  in  the  content  and  difficulty 
of  troubleshooting  tasks  between  two  doppler 
radars.  The  AN/APN-147,  which  is  used  on  the 
C-130  and  C- 141,  has  approximately  14,000  shop 
replaceable  units  (SRU)  whereas  the  inertial 
doppler  navigation  equipment  (IDNE)  on  the  C-5 
has  only  28.  This  lack  of  correspondence  of  func- 
tions across  electronic  hardware  makes  it  difficult 
to  generalize  from  results  of  PM  from  one  elec- 
tronic hardware  to  another.  One  exception  is  in 
the  area  of  general  test  equipment  which  may  be 
used  in  performing  maintenance  tasks  across  many 
hardware  subsystems. 

The  examples  given  are  characteristic  of  many 
of  the  electronic  maintenance  AFSCs.  Similar 
problems  in  complexity  of  maintenance  functions 
and  tasks  arc  found  in  mechanical  hardware,  but 
to  a lesser  degree. 

V.  DEVELOPMENT  OF  PM  AND  SYMBOLIC 
SUBSTITUTES  FOR  PM 

Starting  in  1969,  AS  supported  a modest 
program  to  provide  the  Air  Force  with  the  neces- 


sary tools  for  measuring  the  ability  of  maintenance 
personnel  to  perform  the  key  tasks  of  their  jobs. 
The  scope  of  this  work  was  limited  to  the  main- 
tenance of  electronic  hardware  at  the  organiza- 
tional and  intermediate  levels  (see  shaded  portion 
of  Figure  1 ).  This  program  had  two  objectives:  (a) 
to  develop  a model  battery  of  JTPT  together  with 
appropriate  scoring  schemes  for  the  measurement 
of  the  task  performance  ability  of  electronic  main- 
tenance personnel  (an  effort  was  to  be  made  for 
the  development  of  JTPT  which  could  be  easily 
administered),  and  (b)  using  the  JTPT  of  this 
battery  as  criteria,  to  develop  and  try  out  a series 
of  paper-and-pencil  symbolic  substitute  tests  that 
would  hopefully  have  high  empirical  validity. 

Criterion  Referenced  Job  Task 
Performance  Tests 

A model  battery  of  48  criterion  referenced 
JTPT  and  a test  administrator’s  handbook  were 
developed  for  measuring  ability  to  perform  elec- 
tronic maintenance  tasks.  Copies  of  the  actual 
instructions  for  test  subjects  together  with  the  test 
administrator’s  handbook  arc  available  from  the 
Defense  Documentation  Center  (DDC)  as 
AFHRDTR- 74-57(11)  Part  11  (Shriver,  Hayes,  & 
Hufhand,  1975).  The  test  administrator’s  hand- 
book was  developed  with  step-by-step  detailed 
instructions  so  that  an  individual  with  a minimum 
of  electronic  maintenance  experience  can 
administer  the  tests. 

The  battery  includes  separate  tests  for  the  fol- 
lowing classes  of  job  activities:  (a)  equipment 
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checkout,  (b)  alignment/calibration,  (c)  removal/ 
replacement,  (d)  soldering,  (e)  use  of  general  and 
special  test  equipment,  and  (0  troubleshooting. 
The  Doppler  Radar  AN/APN-147  and  its  Com- 
puter AN/ASN-35  were  selected  as  a typical  elec- 
tronic system.  This  system  was  used  as  the  test  bed 
for  this  model  battery.  The  soldering  and  general 
test  equipment  JTPT  are  applicable  to  all  elec- 
tronic technicians.  The  other  tests  of  the  battery 
apply  to  technicians  concerned  with  this  specific 
doppler  radar  system.  A detailed  description  of  the 
development  and  tryout  of  these  JTPT  is  given  in 
AFHRL-TR-74-570I)  Part  I (Shriver  & Foley, 
1974a).  Each  class  of  activity  for  which  JTPT  was 
developed  contains  its  individual  mix  of  behaviors, 
but  it  is  not  mutually  exclusive.  As  indicated  in 
Figure  5 and  Table  1,  a four-level  hierarchy  of 
dependencies  exists  among  them. 

After  considering  product,  process,  and  time  as 
to  their  appropriateness  for  scoring  the  results  for 
each  activity,  it  was  decided  that  a test  subject  had 
not  reached  criterion  until  he  had  produced  a 
complete,  satisfactory  product.  This  was  a go, 
no-go  criterion. 

Table  2 summarizes  the  number  of  tests, 
problems  and  scorable  products  by  class  developed 
for  the  AN/APN-147  and  AN/ASN-35.  The  simple 
addition  of  numbers  shown  in  Table  2 indicates 
that  there  are  48  tests,  81  problems,  and  133 
scorable  products.  But,  these  numbers  tell  us 
nothing  in  terms  of  the  content  of  the  tests.  To 
say  that  one  test  subject  accomplished  100 
scorable  products  while  another  accomplished  90, 
tells  us  nothing  about  the  job  readiness  of  these 
individuals  or  that  one  is  better  than  the  other. 
The  varieties  of  scorable  products  are  so  diverse 
that  any  combination  of  them,  without  regard  to 
what  they  represent,  is  meaningless.  The  only 
meaningful  presentation  of  such  information  must 
be  in  terms  of  a profile  designed  to  attach  meaning 
to  such  numbers.  A sample  of  such  a profile  is 
shown  in  Figure  6. 


This  profile  is  not  presented  as  the  final  so- 
lution to  the  profile  problem  for  JTPT  for  elec- 
tronic maintenance.  It  does  contain  most  of  the 
important  information  regarding  a test  subject’s 
success  on  the  full  range  of  tests.  It  gives  a mean- 
ingful picture  of  the  subject’s  job  task  abilities  as 
measured  by  the  test  battery,  indicating  the 
subject’s  strengths  and  weaknesses. 

An  examination  of  the  profile  (Figure  6) 
indicates  that  most  of  the  tests  in  this  battery 
contain  only  one  problem.  For  example,  there  are 
two  checkout  tests,  having  one  problem  each  and 
there  are  eleven  troubleshooting  tests  having  one 
problem  each.  There  arc  two  soldering  tests;  one 
has  two  problems  and  the  other  has  three.  The 
voltohnimeter  (VOM)  test  has  20  problems. 

The  subject  receives  no  “credit”  for  a problem 
unless  he  obtains  all  of  the  expected  products.  No 
attempt  is  made  to  combine  these  scores  in  terms 
of  meaningless  numbers. 

The  hierarchy  of  dependencies  discussed  pre- 
viously (Figure  5)  has  implication  for  the  order  in 
which  tests  are  administered,  as  well  as  for 
diagnostics.  For  example,  since  troubleshooting 
includes  the  use  of  test  equipment  and  othc' 
activities  in  the  hierarchy,  logic  would  dictate  that 
in  most  training  situations  the  administration  of 
the  tests  for  the  sub-activities  would  precede  the 
troubleshooting  tests  and  that  a test  subject  would 
not  be  permitted  to  take  the  troubleshooting  tests 
until  he  had  passed  these  other  subtests.  Under 
some  circumstances,  one  may  wish  to  reverse  the 
process.  A subject  who  successfully  completes 
selected  troubleshooting  or  alignment  tests  can  be 
assumed  to  be  proficient  in  his  use  of  test  equip- 
ment and  checkout  procedures.  These  depend- 
encies are  displayed  on  the  left-hand  side  of  the 
profile  (Figure  6). 

Due  to  the  unavailability  of  a sufficient  number 
of  experienced  test  subjects  at  the  time  of  the 
tryout  of  the  JTPT  battery,  the  tryout  was  not  as 
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Figure  6.  A profile  for  displaying  the  results  obtained  by  an  individual  subject  from  a battery  of  Job  Task 
Performance  Tests  concerning  an  electronic  system  - the  AN/APN-147  and  the  AN/ASN-35.  This  represents 
the  profile  of  an  individual  who  has  successfuDy  completed  most  of  the  battery. 
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extensive  as  planned.  The  limited  tryout  did  indi- 
cate that  the  tests  as  developed  are  administra- 
tively feasible.  Their  continued  use,  no  doubt, 
would  result  in  further  modifications  and 
improvements. 

Development  of  Symbolic  Substitutes 

There  is  no  doubt  that  a battery  of  JTPT  would 
require  more  training  and  on-the-job  time  of  the 
test  subjects,  more  equipment,  and  specially 
trained  test  administrators.  Therefore,  the  avail- 
ability of  empirically  valid  symbolic  substitute 
tests  would  be  highly  desirable.  Even  though 
previous  attempts  to  develop  such  tests  as  the  Tab 
Test  (Crowder,  Morrison,  & Demarec,  1954)  had 
failed,  it  was  our  opinion  that  much  more  work 
could  be  done  to  improve  symbolic  maintenance 
tests  as  substitutes  for  JTPT.  It  was  hypothesized 
that  higher  correlations  possibly  could  be  obtained 
by  a different  approach  to  the  development  of 
symbolic  tests.  A study  of  the  Tab  Tests  (Crowder 
et  al..  1954,  sec  Table  1)  indicated  that  the  JTPT 
used  as  the  criterion  measures  contained  many 
distractions  and  interruptions  to  the  subject’s 
troubleshooting  strategy  (cognitive  process);  such 
as,  using  test  equipment  to  obtain  test  point 
information.  In  addition  to  such  interruptions  to 
the  cognitive  process,  the  subject  can  obtain  faulty 
test  point  information  by  the  improper  use  of  his 
test  equipment.  In  the  symbolic  substitute  Tab 
Tests,  all  of  these  potential  pitfalls  of  the  actual 
task  were  avoided.  The  subject  was  given  a printed 
test  point  readout.  It  was  hypothesized  that  the 
injection  of  job  equivalent  pitfalls  into  symbolic 
substitutes  possibly  would  increase  their  empirical 
validity. 


Based  on  these  hypotheses,  a battery  of  sym- 
bolic tests  was  developed  under  contract  with  the 
Matrix  Research  Company  of  Falls  Church, 
Virginia.  A companion  graphic  symbolic  test  was 
developed  for  each  of  the  job  activities  for  which  a 
criterion  referenced  JTPT  had  previously  been 
developed.  Based  on  two  limited  validations,  all  of 
the  graphic  symbolic  tests,  with  the  exception  of 
the  symbolic  test  for  soldering,  indicated  sufficient 
promise  to  justify  further  consideration  and  refine- 
ment. Table  3 indicates  the  correlations  obtained 
from  these  validations.  Due  to  a shortage  of  avail- 
able subjects,  the  number  of  pairs  of  subjects  was 
extremely  small.  All  of  these  promising  graphic 
symbolic  tests,  therefore,  must  be  given  more 
extensive  validations  using  larger  numbers  of 
experienced  subjects. 

The  validation  of  any  such  symbolic  test 
requires  the  administration  of  a companion  JTPT 
as  a validation  criterion.  As  a result,  a validation  is 
an  expensive  process  in  terms  of  equipment  and 
experienced  manpower.  Tire  troubleshooting 
symbolic  tests  require  the  most  extensive  refine- 
ment. Several  suggestions  are  made  for  improving 
their  empirical  validity.  A complete  description  of 
these  symbolic  test  efforts  can  be  found  in 
AFHRL-TR-74-57(IU)  (Shriver  & Foley,  1974b). 
An  attempt,  also,  was  made  to  develop  video 
symbolic  substitute  tests,  but  this  efrort  produced 
no  promising  results.  (Shriver.  Hayes.  & Hufhand, 
1974). 

Even  if  graphic  symbolic  substitutes  of  high 
empirical  validity  can  be  produced,  the  use  of 
symbolic  substitutes  will  never,  in  my  opinion, 
dispense  with  the  requirement  for  the  liberal 
administration  of  actual  JTPT  to  maintenance 


Table  .1  Indicates  the  Number  of  Pair*  Used  as  Well 
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personnel.  We  can  never  include  all  aspects  of  an 
actual  performance  of  a task  in  a paper-and-pencil 
symbolic  representation  of  that  task,  but  our  work 
indicates  that  we  can  come  much  closer  than  has 
been  done  in  the  past. 

The  Sampling  Problem 

Timewise,  it  would  be  impossible  to  administer 
a JTPT  to  a maintenance  man  for  every  possible 
task  that  his  hardware  system  might  produce.  This 
world  of  tasks  and  people  must  be  sampled.  The 
model  battery  described  previously  provides  a 
sampling  procedure  based  on  major  task  functions 
such  as  checkout,  align,  adjust,  troubleshoot,  etc. 
But  even  this  sampling  across  possible  tasks  re- 
sulted in  48  tests  and  133  scorable  products  (Table 
2).  It  would  be  impractical  to  give  any  one  test 
subject  all  of  these  48  tests  at  any  one  time. 
Systematic  sampling  schemes  must  be  developed 
across  tests. 

The  purposes  for  which  JTPT  results  are  to  be 
used  should  be  considered  when  developing 
sampling  schemes.  Such  purposes  could  include 
ascertaining  (a)  the  job  task  proficiency  of  an 
individual,  (b)  the  job  effectiveness  of  a training 
program,  and  (c)  the  proficiency  of  a maintenance 
unit.  Each  of  these  purposes  would  require  a 
different  mix  or  mixes  of  tests  and  people.  Some 
suggestions  for  such  samplings  can  be  found  in 
AFHRL-TR-74-57(II)  Part  I (Shriver  & Foley. 
1974a).  But  it  should  be  remembered  that  these 
are  suggestions  that  must  still  be  field  tested. 

In  the  case  of  determining  unit  proficiency, 
some  JTPT  can  be  administered  by  on-line  obser- 
vation of  tasks  which  are  often  repeated  such  as 
checkout.  There  will  always  be  a requirement  for 
off-line  PM  concerning  critical,  but  seldom 
performed  tasks.  Whether  the  JTPT  is  performed 
on-line  or  off-line,  the  test  administrator  must  use 
the  same  objective  scoring  procedures,  the  criteria 
of  success  being  acceptable  products. 


VI.  CONSOLIDATED  DATA  BASE 
TO  SUPPORT  PM 

In  keeping  with  its  man-machine  interface 
orientation,  AFHRL/AS  is  demonstrating  the  tech- 
nical feasibility  of  integrating  five  human  resources 
related  technologies  and  applying  them  during 
weapon  system  development.  This  is  being  accom- 
plished under  project  1959,  “Advanced  System  for 
the  Human  Resources  Support  of  Weapon  System 
Development."  The  five  technologies  are:  (a) 


human  resources  in  design  trade  offs,  (b)  main- 
tenance manpower  modeling,  (c)  job  performance 
aids,  (d)  instructional  system  design,  and  (e) 
system  ownership  costing. 

One  objective  of  this  program  is  to  determine 
the  data  input  requirements  for  and  prepare  speci- 
fications for  a consolidated  maintenance  task 
identification  and  analysis  data  base  which  will 
support  the  integrated  application  of  these  five 
technologies  in  a weapon  system  development 
program.  We  feel  that  such  a consolidated  data 
base  will  contain  most,  if  not  all,  of  the  informa- 
tion which  would  be  required  to  develop  good 
JTPT  provided  the  tests  are  developed  in  keeping 
with  the  technology  described  in  this  paper.  If 
such  a data  base  is  demonstrated  to  be  technically 
feasible  and  if  it  is  routinely  made  a requirement 
in  weapon  system  development  contracts,  it  will 
provide  considerable  assistance  in  developing  main- 
tenance performance  tests  for  new  weapon 
systems. 

VII.  INSTITUTIONALIZATION  OF 
NEW  TECHNOLOGIES 

Getting  newly  developed  technologies  such  as 
PM  institutionalized  is  a perennial  problem, 
especially,  when  a technology  requires  funda- 
mental changes  in  long  existing  programs,  proce- 
dures. and  attitudes  of  entrenched  establishments. 
AS  has  been  involved  in  tire  implementation  of 
several  well  developed  and  documented  tech- 
nologies, such  as  job  performance  aids  and  instruc- 
tional systems  design  (ISD)  including  programmed 
instruction  and  job  (task)  oriented  training.  These 
experiences  have  indicated  that  it  is  extremely 
difficult  to  maintain  the  integrity  of  a technology 
during  its  so-called  implementation.  Operational 
organizations  invariably  attempt  to  implement  a 
much  “watered  down”  version  of  the  technology 
and  consequently  obtain  much  “watered  down” 
results.  In  some  cases  only  cosmetic  changes  to 
existing  programs  are  reported  as  implementations. 
Currently  it  requires  years  of  persistent  effort  on 
the  part  of  the  research  community  to  get  a 
technology  properly  institutionalized. 

A mechanism  mast  be  developed  for  the  timely 
institutionalization  of  each  new  technology  which 
will  ensure  its  integrity.  A mechanism  for  the 
orderly  implementation  of  technologies,  similar  to 
that  used  for  new  weapons  systems,  is  re- 
commended. Such  a mechanism  must  make 
efficient  and  effective  use  of  the  “know-how"  of 
the  developers  of  the  technology  and  make  them 
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responsible  and  accountable  for  its  implementa- 
tion. A new  technology  should  not  be  turned  over 
to  a using  command  for  its  operation  until  it  is  in 
place,  “debugged"  and  operational -just  as  a new 
weapons  system  is  not  turned  over  to  an 
operational  command  until  it  has  been 
“debugged”  and  proven  to  be  ready  for 
operational  use. 


VIII.  PROPOSED  PM  RAD  EFFORTS 
FOR  MAINTENANCE 

Excessive  maintenance  costs  are  never  going  to 
be  reduced  as  long  as  we  don’t  have  JTPT  and/or 
empirically  valid  symbolic  substitutes  to  ascertain 
how  efficiently  maintenance  men  perform  the 
tasks  of  their  jobs.  In  my  opinion,  the  lack  of  such 
measures  of  maintenance  performance  is  a most 
serious  deficiency  in  DOD.  As  such,  R&D  in  this 
area  should  have  an  extremely  high  priority. 

Areas  for  R&D  Concentration 

For  a long  range  R&D  effort,  five  general  areas 
of  concentration  are  recommended;  namely  JTPT 
and  matching  symbolic  substitute  tests  for  elec- 
tronic maintenance,  JTPT  and  matching  symbolic 
substitute  tests  for  mechanical  maintenance,  and 
aptitude  tests  based  on  PM.  The  development  and 
field  tryout  of  a JTPT  must  precede  the  develop- 
ment of  its  symbolic  substitute.  The  work  on 
JTPT  batteries  for  both  electronic  and  mechanical 
maintenance  should  be  started  as  soon  as  possible. 
The  work  on  aptitude  tests  should  not  be  started 
until  JTPT  batteries  and  the  symbolic  substitute 
tests  have  been  completely  field  tested.  More 
information  concerning  these  areas  of 
concentration  follows: 

1 .  Refinement  of  Model  JTPT  Battery  ( Elec- 
tronic Maintenance).  The  already  available  model 
JTPT  Battery  (Shriver,  Hayes,  & Hufhand,  1975) 
should  be  given  a large  scale  field  tryout.  (Since 
the  AB328X4  Avionic  Inertial  and  Radar  Naviga- 
tion Systems  Specialist  Course,  which  includes  the 
AN/APN-147  and  the  AN/ASN-35,  does  not 
emphasize  the  mastery  of  job  tasks,  the  equip- 
ment-specific tests  of  this  battery  cannot  be  used 
in  the  formal  course.)  One  thrust  of  this  effort 
should  be  to  further  refine  the  battery  including 
its  administrative  procedures.  A second  thrust 
should  be  the  development  of  sampling  strategies 
which  would  be  appropriate  for  determining  the 
effectiveness  of  training  programs  and  both  indi- 
vidual and  unit  proficiency  as  discussed  earlier 
under  PM  problems.  This  effort  would  require 


approximately  two  professional  man-years  plus  the 
use  of  maintenance  specialists  as  test  admin- 
istrators from  the  appropriate  maintenance 
specialties.  If  it  is  necessary  to  select  a system 
other  than  the  AN/APN-147-AN/AJN-35  combina- 
tion, this  work  would  require  approximately  four 
professional  man-years. 

2.  Refinement  of  Symbolic  Substitutes  (Elec- 
tronic Maintenance).  As  previously  indicated,  a 
number  of  symbolic  substitutes  for  JTPT  were 
developed  and  given  a limited  tryout.  Table  3 
indicated  that  some  of  the  symbolic  tests  show 
promising  empirical  validity.  These  promising 
symbolic  tests  must  be  more  thoroughly  refined 
and  validated.  In  addition,  further  exploratory 
development  is  required  for  symbolic  substitute 
tests  for  troubleshooting  tasks  in  keeping  with 
recommendations  made  in  AFHRL-TR-74-57(lll) 
(Shriver  & Foley,  1974b).  This  effort  would 
require  between  three  and  four  professional 
man-years  plus  the  use  of  maintenance  specialists 
as  test  administrators  and  test  subjects  from  the 
appropriate  maintenance  specialties. 

3.  Development  of  Model  JTPT  Battery 
(Mechanical  Maintenance).  A model  JTPT  battery 
similar  to  the  model  battery  for  electronic  main- 
tenance described  previously  should  be  developed 
for  a typical  mechanical  subsystem  such  as  a jet 
engine  or  tank  engine  oovering  both  the 
organizational  and  intermediate  levels  of  main- 
tenance. This  model  should  be  thoroughly  field 
tested.  Sampling  strategies  as  indicated  for  the 
electronic  battery  should  also  be  developed.  This 
effort  will  require  approximately  four  professional 
man-years  plus  the  use  of  maintenance  men  from 
the  appropriate  maintenance  specialties  as  test 
administrators  and  test  subjects. 

4.  Development  of  Symbolic  Substitutes 
(Mechanical  Maintenance).  An  attempt  should  be 
made  to  develop  symbolic  substitute  tests  with 
high  empirical  validity  after  the  model  JTPT 
battery  is  available  for  mechanical  maintenance. 
The  same  contractor  should  develop  these 
symbolics  as  developed  the  JTPT  battery.  A very 
rough  estimate  for  accomplishing  this  symbolic 
effort  would  be  four  professional  man-years. 

5.  Job  Aptitude  Test  Research  Based  on 
Results  on  JTPT.  R&D  plans  should  be  made  to 
utilize  the  results  of  JTPT  and  symbolic  substitute 
tests  for  standardizing  military  aptitude  indices 
obtained  from  the  Armed  Services  Vocational 
Aptitude  Battery  (ASVAB).  As  a first  step,  the 
military  aptitude  scores  of  all  tests  subjects  used 
for  the  tryouts  in  the  proposed  JTPT  R&D  should 
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be  recorded.  In  addition,  such  aptitude  scores 
should  be  obtained  during  any  school  or  field 
administration  of  JTPT  or  symbolic  substitutes. 
When  sufficient  data  are  obtained,  the  degree  of 
relationship  between  JTPT  results  and  various 
aptitude  indices  should  be  obtained.  iMter,  when  a 
sufficient  number  of  JTPT  are  used  in  the  field,  a 
formal  R&I)  project  should  he  initiated  to  modify 
the  ASVAB  to  directly  reflect  job  success  as 
measured  by  JTPT. 

R&D  Strategy.  Probably  the  most  cost-effective 
approach  for  PM  for  both  electronic  and  me- 
chanical maintenance  would  be  to  concentrate  on 
the  development  and  refinement  of  JTPT  on  use 
of  key  test  equipments  prior  to  proceeding  with 
the  other  task  functions  of  the  proposed  model 
test  batteries.  As  indicated  in  Figure  5,  the  use  of 
general  test  equipment  is  a prerequisite  to  main- 
tenance task  functions  such  as  alignment,  calibra- 
tion and  troubleshooting.  In  addition,  general  test 
equipments  usually  have  wide  usage  in  such  task 
functions  across  many  hardware  systems  and  there 
are  substantial  amounts  of  data  which  indicate 
that  many  maintenance  men  are  weak  in  their  test 
equipment  ability.  So,  a general  improvement  in 
ability  to  use  lest  equipment  is  an  important  and 
necessary  factor  for  the  general  improvement  of 
several  maintenance  task  functions.  I would 
slrongjy  recommend,  therefore,  that  the  early 
concentration  for  the  proposed  model  test 
batteries  in  this  area.  Fach  PM  development  for  a 
test  equipment  should  be  accompanied  by  the 
development  of  a programmed  training  package 
with  sufficient  practice  frames  for  teaching  the 
mastery  of  all  its  functions.  Basic  models  of  such 
training  packages  lor  12  general  test  equipments 
arc  now  available  (see  Scott  & Joyce,  1975a 
through  19751)  Howe  er.  more  practice  frames 
should  be  included  in  rh  ;se  programs. 


IX.  CLOSINr.  STVn-MPNT 

Maintenance  of  hardware  is  currently  an  ex- 
tremely costly  operation  for  the  IX)D.  High  main- 
tenance cost  is  die  primary  cause  of  high  systems 
ownership  cost.  For  some  electronic  maintenance 
specialties,  nearly  one  year  of  broad  formal 
training  is  given  first  enlistment  personnel.  And 
maintenance  training  generally  is  long  and  costly. 


Even  with  such  lengthy  training,  the  efficiency  of 
maintenance  could  be  greatly  improved.  Improved 
job  instructions  and  information  as  well  as  in- 
creased use  of  job  (task)  oriented  training  have 
great  potential  for  decreasing  maintenance  training 
time  and  improving  the  job  performance  of  main- 
tenance tasks.  But  to  realize  such  potential,  the 
criteria  for  the  personnel  system  (selection, 
training,  assignment  and  promotion)  for  main- 
tenance personnel  must  be  shifted  to  the 
demonstrated  ability  to  perform  the  tasks  of  their 
jobs.  (The  current  criteria  emphasize  the  ability  to 
obtain  high  scores  on  paper-and-pencil  dieory  and 
job  knowledge  tests.) 

In  this  paper,  I have  discussed  what  1 think  are 
the  important  aspects  of  the  criterion  problem  as 
it  applies  to  the  measurement  of  ability  to  perform 
maintenance  tasks  in  training  and  on  the  job.  Our 
objective  in  its  solution  is  to  get  as  close  to  the  real 
job  as  possible.  When  “on-line”  tasks  occur  often 
enough,  their  structured  observation  may  be 
appropriate.  But  when  such  observations  are  not 
appropriate  or  when  tasks  occur  infrequendy,  we 
propose  to  have  the  tasks  performed  “off-line”  in 
a job-like  environment.  Our  approach  to  the 
development  of  such  measures  was  started  with  an 
analysis  of  the  structure  maintenance  of  the  man/ 
hardware  interface.  Based  on  the  results  of  this 
analysis,  we  developed  a model  test  battery  of 
JTPT  for  electronic  maintenance.  Using  this  model 
as  the  criterion,  we  also  developed  batteries  of 
graphic  and  video  symbolic  substitute  tests. 
Several  of  the  graphic  symbolics  have  indicated 
respectable  empirical  validities  but  require  more 
refinement  and  tryout.  Our  attempts  to  develop 
video  symbolics  were  unsuccessful. 

I have  recommended  a research  program  based 
on  what  we  have  already  accom plished.  This 
includes  die  development  of  a model  battery  of 
JTPT  together  with  symbolic  substitutes  for  main- 
tenance tasks  generated  by  a typical  mechanical 
hardware.  I have,  also,  discussed  briefly  the 
perennial  problem  of  getting  new  technologies 
such  as  JTPT  implemented.  There  is  definitely  a 
requirement  for  a structured  mechanism  which 
will  guarantee  the  orderly  institutionalization  of 
such  technologies  as  well  as  their  integrity  during 
the  institutionalization  process. 
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